In a non-orthogonal multiple access (NOMA) technique, overlapping coding is formed with respect to multiple users and overlaps with a power domain and an overlapped signal is separated by using successive interference cancellation (SIC) at each receiving side. The NOMA technique can increase frequency efficiency by using a new domain such as the power domain. However, the NOMA technique may have a shadow zone similarly to other wireless access technologies and a problem in that the NOMA technique cannot provide sufficient coverage or a data rate desired by a user. In order to solve the problem, using the NOMA technique together with a relay technique becomes a natural extension. The relay technique can resolve the shadow zone problem, increase cell coverage, and improve the frequency efficiency. In particular, small cell build-up using a wireless relay can significantly reduce network build-up and maintenance cost. Further, a cooperative relay technique using multiple users in an idle state as a relay has an advantage in that cost can be minimized while maximizing an advantage of the relay technique.
However, when the wireless relay or the wireless small cell is used, in general, a base station and the relay do not perform simultaneous transmission in the same band due to mutual interference and perform transmission by using different frequency bands or at different time. Orthogonal transmission between the base station and the relay can deteriorate the frequency efficiency. The deterioration problem of the frequency efficiency due to the orthogonal transmission between the base station and the relay occurs even in the case of just applying the NOMA technique to a wireless relay system or a wireless small cell system.